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Abstract 

In an effort to foster research integrity, the National Institutes of Health and the National Science 
Foundation mandate education of all trainees in the responsible conduct of research (RCR). 
Nevertheless, recent studies suggest that rates of questionable research practices and scientific 
misconduct are both high and considerably underreported. In part, this may be due to the fact 
that some ethical norms (e.g., authorship assignment) are far from clear and researchers are 
unsure how to respond to perceived misconduct. With funding from the U.S. Office of Research 
Integrity (ORI), we convened four panels of experts to develop a consensus on the overarching 
goals and teaching content of RCR instruction. Our panelists recommended nine overarching 
objectives for RCR instruction that require us to rethink common modes of instruction, and 
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they identified issues and standards that should be covered within controversial areas such as 
authorship assignment and whistle-blowing. Additionally, our experts recommended two new 
core areas for RCR instruction: The social responsibilities of scientists and current topics in RCR. 

Keywords-, responsible conduct of research, research integrity, research ethics, instruction, training, 
assessment 


Introduction 

Responsible Conduct of Research (RCR) education and training too often emphasize rules 
like “Do not falsify data” or “Do not plagiarize.” These are simple extrapolations of what most 
researchers learned in kindergarten: lying and stealing are wrong. Reminding researchers of 
such rules involves stating the obvious, with the result that RCR education and training may be 
perceived as boring, unnecessary, and ineffective. 

However, not all issues in research ethics are so clear-cut. In a survey by Martinson, 

Anderson, & de Vries (2005) of over 1,700 researchers, 33% reported engaging in so-called 
“questionable research practices” such as dropping data points from analyses based on a 
hunch or inappropriately assigning authorship. The example of inappropriate authorship is 
particularly instructive. First, practices for assigning authorship vary across disciplines (Steneck, 
2004). Second, even in a discipline such as medicine, in which international standards have 
been published (International Committee of Medical Journal Editors, 2007), authorship 
assignment has not become standardized. A recent review of 234 biomedical journals found 
that 41% gave no guidance about authorship and only 19% were based on the current criteria 
of the International Committee of Medical Journal Editors (Wager, 2007). Uncertainty about 
criteria helps to explain the high rates at which researchers admit to assigning authorship in a 
questionable manner. Yet, given a lack of standardized criteria within professions, even RCR 
instructors are uncertain what should be taught in the area of authorship. 

While rates of strict research misconduct (data falsification, fabrication, or plagiarism) are much 
lower than rates of questionable practices, they are also higher than many might assume. A 
survey by the U.S. Office of Research Integrity (ORI) of researchers holding funding from the 
National Institutes of Health (NIH) at 605 different institutions, inquired into the number of 
times researchers had observed suspected research misconduct in their own departments over 
the previous three academic years (Titus, Wells, & Rhoades, 2008). A total of 2,212 researchers 
completed the survey (yielding a 51% response rate); they reported observing a total of 201 
instances. By extrapolating this rate of observed suspected misconduct — assuming that the 49% 
who did not respond observed no instances of misconduct — the authors estimated that there 
are more than 2,300 observations of likely misconduct per year in research funded by the U.S. 
Department of Health and Human Services (DHHS). 

Given that ORI receives an average of only 24 institutional investigation reports per year 
(approximately 1% of the estimated incidences observed), these numbers suggest the need for 
RCR education and training — not only to reduce rates of misconduct, but also to provide 
guidance to researchers in how to respond to observed misconduct. Yet this topic is also 
controversial. Real-world decisions regarding whistle-blowing are often far more complex (Smith, 
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2006) and their consequences far more devastating (Couzin, 2006) than ethics textbooks suggest. 
While it is not sufficient for RCR instructors to remind people of a duty to report misconduct, it 
is unclear precisely what content or standards should be taught. 

In 2000, ORI identified nine cores areas that RCR courses should address: (1) data acquisition, 
management, sharing, and ownership; (2) mentor/trainee responsibilities; (3) publication 
practices and responsible authorship; (4) peer review; (5) collaborative science; (6) human 
subjects; (7) research involving animals; (8) research misconduct; and (9) conflict of interest and 
commitment. While these core areas provide a useful initial framework, there is no evidence 
of professional consensus that ORI’s list includes the most important areas of RCR, nor what 
content should be taught and assessed within the core areas (Steneck & Bulger, 2007). For 
example. Pimple (2002) has recommended approaching RCR through the lens of six domains, 
some of which overlap with the nine core areas, and some of which extend into new areas 
such as social responsibilities (including fiscal responsibilities, advocacy by researchers, and 
environmental impact). 

RCR trainers may also have different goals in mind: to convey knowledge of right and wrong; 
to foster professional virtues; to inculcate values that support good science, to raise awareness of 
ethical issues; to motivate people to do what is right; and — most ambitiously — to improve 
behavior (DuBois, Ciesla, & Voss, 2001). The behavioral goal is probably the most widely 
proffered — even if controversial — insofar as ethics instructors frequently begin courses, 
textbooks, or funding proposals by citing instances of scientific misbehavior, thus implying that 
RCR training can help prevent such events. In this vein, one leading research administrator 
writes, “the value of . . . RCR education from an administrative perspective can be summed up 
in the oft-used adage, an ounce of prevention is worth of pound of cure’ (Vasgird, 2007, p. 835). 

Two studies examined the content and goals of RCR education and training. In 2005, Heitman 
and Bulger published a content analysis of 20 RCR textbooks. Content reflected each of ORFs 
core areas and became more comprehensive after ORI published its policy on RCR instruction 
in 2000. The authors also identified gaps in the core areas of compliance, ethics of lab safety, 
institutional responsibilities, and the role of scientists in society (Heitman & Bulger, 2005). 
Kalichman and Plemmons (2007) studied the goals of existing education and training programs. 
They conducted interviews with 50 instructors and identified over 50 distinct goals pertaining to 
knowledge, skills, attitudes and behavior. They found that actual educational goals varied widely 
across instructors. 

These two studies reinforce the need to pursue consensus on RCR instruction. On the one hand, 
important gaps appear to exist in RCR textbooks (e.g., institutional responsibilities and the role 
of scientists in society), while on the other, the study of actual education and training programs 
identified over 50 distinct educational goals, which varied widely across instructors. Add this to 
the vagaries surrounding authorship and whistle-blowing, and a muddy picture of the goals and 
content of RCR instruction emerges. 

Whereas these previous studies examined the goals or content of existing RCR education and 
training programs and materials, our project sought to establish a consensus among experts on 
what RCR education and training shouldXook like. We addressed four specific questions: 
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1. What should be the overarching goals of RCR training (e.g., knowledge, problem-solving 
skills, or virtue)? 

2. Are the nine core areas of RCR instruction identified by ORI complete, or should additional 
core areas be addressed? 

3. Within the core areas, what specific content should be taught? 

4. What objectives and content should be assessed? 

Methods: Delphi Expert Panels 

About Delphi Consensus Panels 

One way of developing recommendations for a field is to convene a diverse panel of experts to 
engage significant questions. Such an approach is regularly used by the U.S. National Academies 
of Science to address questions in the fields of engineering, medicine, and science. With funding 
from ORI we used an online Delphi panel process to foster an expert consensus. Delphi panels 
involve administering a questionnaire to groups of individuals across several rounds with the 
aim of identifying shared evaluations or recommendations (Ferguson, 2000). Key elements of 
the Delphi process are a structured flow of information, controlled feedback to participants, 
statistical analysis of responses, and participant anonymity. Interactions among panel members 
are controlled by a coordinator, who filters feedback and organizes data for subsequent 
presentation in the next round. The Delphi method maximizes the benefits of group decision- 
making while the anonymity of the process minimizes limitations such as domineering group 
members, personality conflicts, or groupthink (Delbecq, Van de Ven, & Gustafson, 1975). Other 
advantages to an online Delphi method include its relatively inexpensive cost and convenience 
for participants, who can access the survey at any time of day. 

Delphi Panel Procedures 

Because few people possess expertise in all areas of RCR, we formed four separate expert panels. 
Each panel worked independently and simultaneously. Our Delphi process involved multiple 
rounds of questioning. Round 1 consisted of an open-response format. Panelists were directed to 
one of four websites corresponding to their panel assignment(s), where responses were collected 
in text-boxes. 

Our panelists were asked: (1) What should be the overarching educational objectives 

of RCR instruction; and (2) Are the nine core areas of RCR instruction complete, or should new 
core areas be addressed within RCR instruction? 

Scientific Data panelists were asked: Within RCR instructional programs, what specific topics 
should be taught and assessed in the areas of: (1) Data acquisition, management, sharing and 
ownership; and (2) Research misconduct? 

Scientific Relationships panelists were asked: Within RCR instructional programs, what specific 
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topics should be taught and assessed in the core areas of: (1) Mentor/trainee responsibilities; (2) 
Collaborative science; and (3) Conflicts of interest and commitment? 

Scientific Publications panelists were asked: Within RCR instructional programs, what specific 
topics should be taught and assessed in the core areas of (1) Publication practices and responsible 
authorship; and (2) Peer review? 

We excluded from our project two of ORPs nine core areas for RCR instruction: human subjects 
and animals. There were several reasons for this: (1) Institutional Review Boards and Institutional 
Animal Care and Use Committees typically mandate ethics training that is separate from general 
RCR training; (2) the scope of these core areas is very large (thus excluding them made the 
project more manageable); (3) it appears that significant consensus exists on what needs to be 
covered in such courses; and (4) these areas do not comprise ORI’s primary areas of focus for 
education and oversight. 

After all participants had completed round 1, their responses were carefully condensed, re-worded 
and organized into topics and subtopics to enhance clarity and prevent redundancy. Round 
2 involved presenting panelists with the revised lists of topics they had generated and asking 
them to rate on a four-point scale the importance of teaching each topic in an RCR course: 1 = 
Unimportant, 2 = Less important, 3 = Important, and 4 = Very important. Panelists were also 
asked to make additional comments about the wording or clarity of each item. 

Topics receiving a vote of “Important’ or “Very Important’ from at least two-thirds of panelists 
were deemed to meet consensus criteria and were presented to panelists in the next round, 
after they were revised according to the panelists’ comments. Topics not meeting consensus are 
displayed in the tables below, with their corresponding consensus values and mean scores. 

Round 3 added to Round 2 by re-asking panelists the importance of teaching item, and also 
asked panelists to rate the importance of assessing each item within an RCR course. Assessment 
rankings followed the same four-point scale used in the previous round. We asked separately 
about the importance of teaching and of assessing goals and content because we believed 
instructors might think some material is worth teaching without the need to assess learning (e.g., 
historical cases taught simply to provide context). 

In each round, we asked panelists to prescind from whether it is fieasible to assess a goal or topic 
and to focus on the importance alone. 

Recruitment 

We recruited experts for our panels during October and November 2006. The three rounds 
were conducted from November 2006 through June 2007. Recruitment began with: (1) a 
literature search to identify authors actively researching and publishing in RCR; (2) a review of 
ORI Annual Reports from 2000 through 2005 to identify those who received ORI contracts 
and grants; and (3) a review of recent research administrative and RCR conference programs 
to identify those who had presented on relevant topics. Based on these activities we generated 
a list of experts with overarching knowledge of RCR, many of whom provide RCR training. 
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Additionally, we produced a list of Chief Research Officers, scientific journal editors, and pre- 
and post-doctoral trainees who we believed would be interested in participating in our consensus 
project. From the resulting list of possible panelists, the Project Director, in consultation with 
ORI, selected those who were both qualified to serve on a particular panel and who represented 
diverse backgrounds. Recruitment letters were sent to these individuals, asking them to volunteer 
without compensation for a total of 1.5 hours (30 minutes for each round) per panel, over 
approximately nine months. Those who declined participation, but represented a subgroup of 
interest, were asked to provide a recommendation for another possible participant. 

Overall, 4 1 individuals served as panelists on either one or two panels, participating in at least 
two of three rounds on any particular panel. (Individuals selected to serve on the Objectives 
panel were also asked to serve on one of the remaining three panels — Scientific Data, Scientific 
Relationships, and Scientific Publications.) The project retained nearly all experts across three 
rounds of questionnaires. The Objectives panel had 18 total panelists, with 16 to 18 panelists 
participating in any given round. The Scientific Data panel had 13 panelists with 12 to 13 
participating per round; Scientific Relationships had 14 panelists with 12 to 14 participating per 
round; and Scientific Publications had 13 panelists with 10 to 12 participating per round. 

Compliance 

This project was presented to the Institutional Review Board at Saint Louis University, which 
concurred with the project director that the project did not constitute research because it was 
aimed at producing a consensus among experts rather than generalizable knowledge. The 
participants were experts serving on a panel seeking consensus on recommendations (akin to 
serving on a committee of the National Academies of Sciences). 


Results 

We defined a consensus as two-thirds of panelists supporting a rating of important or very 
important. What follows are highlights of our findings. 

New Core Areas 

The Objectives panel initially proposed several possible new core areas of RCR instruction. These 
are listed in Table 1 . The panel reached a consensus on two new core areas: Social Responsibilities 
of Researchers and Current Issues in RCR Because consensus on these new core areas only emerged 
through the process itself, the panelists were unable to propose content for these new areas, 
as they did with the pre-existing ORI-recommended core areas. However, some of our expert 
panelists have independently described in publications possible content that could be covered 
under such headings (Bulger & Heitman, 2007; Kalichman, 2002; Pimple, 2002). For example, 
under “social responsibilities,” Pimple (2002) includes research priorities, fiscal responsibilities, 
public service, public education, advocacy by researchers, environmental impact, and forbidden 
knowledge. Under current topics, a wide variety of issues might be discussed. For example, 
in today’s environment, instructors might want to discuss NIH policy on stem cells or the 
relationship between science and politics. 
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Table 1 


Proposed Additional Core Areas 


Core Area 

Percentage of panelists rating item as 
“important” or “very important” (Mean score) 

Teaching 

Assessing 

1 . The financial and operational responsibilities of Principal 
Investigators 

50 (2.56) 

N/A^ 

2. Social responsibilities of researchers 

89* (3.28) 

47 (2.65) 

3. Historical background in responsible conduct of research 

61 (2.72) 

N/A^ 

4. Current issues in responsible conduct of research 

89* (3.28) 

50 (2.61) 

5. Lab safety and environmental health 

56 (2.72) 

N/A^ 

6. Philosophy of science, including roles of bias and world 
views in science 

39 (2.50) 

N/A^ 


Legend: 

* = Item achieved a “consensus” by receiving a rating of important or very important from two-thirds of panelists 

^ = Not applicable because these items were eliminated after round 2 and their importance of being assessed was not 
measured 


Objectives ofRCR Training 

Table 2 presents nine overarching objectives that the panel agreed should be taught in RCR 
training programs. Panelists supported assessing four of these nine objectives: fostering 
understanding of the importance of RCR and the consequences of misbehavior; examining how 
ethics may go beyond compliance with regulations; fostering sensitivity to ethical issues; and 
developing ethical problem-solving skills. 


Table 2 

Overarching Educational Objectives for RCR Instruction 


Topics (Subtopics indented) 

Percentage of panelists rating 
item as “important” or “very 
important” (Mean score) 

Teaching 

Assessing 

1 . Understand the importance of RCR 

94* (3.59) 

81* (2.94) 

a. Know the history of research, including historical examples of 
research misconduct and unethical conduct 

82* (3.29) 

56 (2.50) 

b. Understand the social context of research 

94* (3.29) 

56 (2.78) 
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Topics (Subtopics indented) 

Percentage of panelists rating 
item as “important” or “very 
important” (Mean score) 

Teaching 

Assessing 

c. Consider consequences of unethical conduct in research for self, 
institution, science, and society 

100* (3.83) 

89* (3.22) 

2. Identify sources of RCR regulations and policies 

83* (3.22) 

50 (2.67) 

a. Federal regulations 

83* (3.22) 

61 (2.78) 

b. State laws 

47 (2.65) 

N/A* 

c. Institutional policies 

78* (3.17) 

61 (2.83) 

3. Examine limitations of RCR regulations and policies and variations in 
standards across fields, institutions, and labs 

83* (3.17) 

61 (2.67) 

a. Understand that regulations permit discretion and creative problem 
solving 

89* (3.28) 

61 (2.72) 

b. Understand that regulations require discretion and creative problem 
solving 

61 (2.94) 

N/A* 

c. Understand that regulations do not cover all ethical responsibilities 

94* (3.50) 

78* (3.11) 

4. Understand key distinctions within the field of RCR 

89* (3.22) 

72* (2.83) 

a. Distinctions within ethics, such as ethically obligatory, prohibited, 
and praiseworthy actions 

47 (2.65) 

N/A* 

b. Distinction between ethical and regulatory duties 

83* (3.33) 

56 (2.67) 

c. Distinction between research misconduct and questionable research 
practices 

83* (3.33) 

67* (3.06) 

5 . Foster research integrity or professional character 

94* (3.65) 

44 (2.61) 

a. Motivate morally good action 

83* (3.22) 

39 (2.56) 

b. Inculcate professional values such as pursuit of truth, honesty, 
intellectual humility 

100* (3.39) 

50 (2.72) 

6. Foster ethical sensitivity or the ability to identify ethical issues in the 
conduct of research 

94* (3.50) 

83* (3.06) 

a. Identify common ethical issues such as those addressed within the 
core areas of RCR 

83* (3.06) 

72* (2.89) 

b. Identify threats to RCR, including pressures in the research 
institution and personal bias 

100* (3.67) 

61 (2.83) 

c. Distinguish between ethical responsibilities in research versus other 
professional activities such as education or clinical care 

50 (2.50) 

N/A* 

7. Develop ethical problem-solving skills 

89* (3.44) 

78* (3.11) 

a. Knowledge of relevant ethical frameworks, theories or principles 

67* (2.72) 

33 (2.39) 

b. Ability to identify key elements of an ethical situation, including 
stakeholders, relevant ethical and legal norms, relevant facts, and 
options 

89* (3.39) 

67* (2.89) 

c. Ability to critically reason using ethical principles or values 

83* (3.39) 

78* (2.94) 
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Topics (Subtopics indented) 

Percentage of panelists rating 
item as “important” or “very 
important” (Mean score) 

Teaching 

Assessing 

d. Ability to identify ethical resources, such as mentors, peers, 

institutional officers, educational resources, and consultation services 

83* (3.28) 

56 (2.72) 

8. Examine ways of preventing ethical problems in research 

89* (3.22) 

61 (2.67) 

9. Provide an open forum for discussion of individual RCR concerns and 
challenges 

83* (3.33) 

50 (2.33) 


Legend: 

* = Item achieved a “consensus” by receiving a rating of important or very important from two-thirds of panelists 

^ = Not applicable because these items were eliminated after round 2 and their importance of being assessed was not 
measured 


Instructional Content 

Within the seven core areas of RCR instruction that we examined, the panels achieved a 
consensus on the importance of teaching 43 main topics (with 0-6 specifications of content 
within each main topic). They supported assessing learning in 21 of these 43 main topics. 

Tables 3-9 provide results from the three panels dedicated to ORI’s core areas of RCR instruction 
(See Appendix A). To illustrate the topics identified, within the core area of publication practices 
and responsible authorship, the panelists identified nine main topics instructors should address: 
the significance of authorship; authorship assignment; inappropriate authorship practices; dealing 
with controversies surrounding authorship; scientific responsibilities of authors; poor publication 
practices; protecting privacy in publications; addressing the study’s ethical compliance within 
articles; and responsible disclosure of scientific information within the popular press. Within 
most of these areas, further subtopics were recommended. For example, under the controversial 
topic of “authorship assignment” our panel arrived at a consensus that RCR courses should 
address: criteria for authorship (including substantial intellectual contribution to the study or 
paper and familiarity with and approval of the final text); the ideal of transparent contributions 
(i.e., descriptions of authors’ roles on the paper); how to deal with multiple authors; the 
appropriateness of discussing authorship at the outset of a project; the purpose and examples of 
acknowledgements versus authorship; and discussion of variations of published standards and 
norms across disciplines. Similarly, within the general topic of “responding to misconduct,” 
panelists identified several specific issues that should be addressed: the responsibilities of 
whistleblowers; the evidence needed to report misconduct; protections for whistleblowers; and 
alternatives to whistle-blowing, including illustrations of good and poor responses to observed 
misconduct. 
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Conclusions 

Based on our review of the literature, our project appears to be the first attempt to convene a 
large group of experts to determine the ideal objectives and content of RCR instruction and 
assessment. 

The project is limited in that it reflects the consensus among specific individuals; were 
different individuals selected, our consensus would likely be different. Moreover, we asked 
experts to consider RCR instruction in general — regardless of the trainee populations. If 
given the opportunity, experts might recommend different educational objectives or topics for 
undergraduate science students versus independent investigators. 

Despite its limitations, our project reflects the consensus of individuals with considerable 
expertise, and ORI is exploring ways to disseminate our findings and recommendations to RCR 
instructors. We believe our results may guide the development and quality improvement of RCR 
education and training programs in several ways. 

First, our results provide instructors with guidance in developing content for RCR curricula. For 
example, our project identified issues and standards that should be addressed across disciplines 
within controversial areas such as authorship attribution and whistle-blowing. This may help 
investigators who feel uncertainty regarding what to teach in the current absence (described 
above) of clear professional standards in some areas like authorship. Certainly the nine major 
topics recommended by the panel on publication practices would provide a useful starting point. 
Moreover, they may empower authors themselves in dealing with others on matters of authorship 
and acknowledgements, particularly in interdisciplinary research, where standards may vary. 

Our experts further proposed two new core areas for RCR instruction: social responsibilities of 
researchers and current issues in RCR. Insofar as some popular training programs have limited their 
treatment of topics to the nine core areas originally proposed by ORI in 2000, this development 
may encourage the teaching of a variety of new topics such as research priorities, fiscal 
responsibilities, advocacy by researchers, or the relationship between science and politics. 

Second, our panels identified important knowledge that should be assessed. Student performance 
on corresponding test items might provide an important measure of how well a course or training 
program fosters relevant knowledge and concepts. 

Third, and most importantly, our project produced a list of objectives for RCR instruction 
that transcends the rote knowledge of regulations and basic societal expectations. For example, 
our panel believed that RCR instruction and education should foster integrity or professional 
character, examine how ethics may exceed compliance with regulations, and develop ethical 
problem-solving skills. These objectives may be described as promoting the development of 
researchers as individuals of integrity, ultimately contributing to the creation of a culture of 
ethics and integrity that goes beyond minimum compliance or risk management. Consider, for 
example, the matter of informed consent. Currently, no regulations require formal assessment of 
the decisional capacity of potential research participants. Yet arguably, an investigator of integrity 
who seeks to transcend mere compliance will recognize the need to ensure that participants 
understand consent information and will have the problem-solving skills to identify what options 
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exist for assessing capacity and to recognize which options best fit particular circumstances. 

These more robust objectives may have far-reaching consequences for how RCR training is 
provided. As Kalichman (2007) observes, “active learning modalities are more effective than 
passive learning . . . We are more likely to internalize and understand new information when 
challenged to do something with it than when someone simply tells us what we ‘should’ know”(p. 
872). Yet formal RCR instruction is often restricted to passive online reading or lecture formats. 
While such instructional formats may foster rote knowledge, we question whether they are well 
suited to fostering professional character, ethical problem-solving skills, and other higher-order 
objectives. In contrast, instructional methods that provide a framework for reasoning through 
complex ethical dilemmas — particularly dilemmas or cases that are relevant to day-to-day 
work — are most effective in fostering moral reasoning and ethical decision-making skills (Bebeau, 
1995; Kligyte, Marcy, Sevier, Godfrey, & Mumford, 2008). 

Given the extensiveness of the content and the complexity of the objectives our experts 
recommended for RCR instruction, it is unlikely that any single education or training 
intervention will meet all of the goals. We recommend that, in addition to offering generic 
instruction on RCR aimed at knowledge of many topics, institutions develop education and 
training programs tailored to the population they serve (Heitman & Bulger, 2005; Kalichman, 
2007). The Council of Graduate Schools’ (2006) recent report on Graduate Education for 
the Responsible Conduct of Research explicitly recommended such a two-tiered approach to 
RCR instruction, noting that “graduate students respond best to RCR training that is directly 
relevant to their experience as graduate students” (p. 25). Such specialized courses might be 
less comprehensive even as they are more relevant and engaging, perhaps focusing more on the 
development of high-order skills as described above. Most importantly, formal RCR instruction 
should be only one component in the overall project of fostering research integrity. Other 
components include: mentoring; the institutional climate; the establishment, communication, 
and enforcement of clear policies by institutions, funding agencies, and journal editors; and codes 
of ethics from professional societies (Adams & Pimple, 2005; Institute of Medicine & National 
Research Council, 2002; Macrina, 2007). In combination with formal RCR training, such efforts 
might eventually achieve some of the loftier goals our panels set for the field of science. 
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Appendix A: 

Tables on Proposed Content for Core Areas of RCR 

Instruction 


Table 3 


Proposed Content for “Data Acquisition, Management, Sharing and Ownership” 


Topic (Subtopics indented) 

Percentage of panelists rating item as 
“important” or “very important” (Mean 
score) 

Teaching 

Assessing 

1 . Ethical values behind the scientific standards for data acquisition, 
management, sharing, and ownership 

92* (3.58) 

75* (2.83) 

a. Confidentiality and privacy 

100* (3.67) 

92* (3.08) 

b. Trustworthiness, honesty, and transparency 

100* (3.75) 

67* (2.92) 

c. Right to property or to prosper from work 

58 (2.67) 

N/A- 

d. Scientific collegiality and virtue of sharing 

100* (3.50) 

67* (2.75) 

e. Value of having regulations and standards 

75* (3.25) 

58 (2.75) 

2. Variations in lab practices — legitimate and illegitimate variations 

92* (3.42) 

58 (2.83) 

3. Data acquisition issues 

100* (3.82) 

82* (3.27) 

a. Informed consent or permission to gather or use data 

100* (3.83) 

83* (3.42) 


The Journal of Research Administration 


Volume XL, Number 1 , 2009 61 


ArticiQS 


b. Sampling and data selection 

100* (3.75) 

83* (3.33) 

c. Verifying and cleaning data 

100* (3.67) 

75* (3.17) 

4. Data storage, protection, and archiving 

92* (3.50) 

67* (2.92) 

a. Techniques for entering, storing, and archiving data 

64 (2.82) 

N/A- 

a. Data storage longevity (how long to save data and what 
format) 

83* (3.17) 

58 (2.67) 

b. Data protection and backup 

92* (3.25) 

67* (2.83) 

c. Unique issues pertaining to special kinds of data, such as 
tissue, DNA, photographic data 

92* (3.33) 

50 (2.83) 

5. Data Sharing 

100* (3.50) 

67* (2.92) 

a. How and when data should be shared, advantages and 
disadvantages 

100* (3.50) 

75* (2.83) 

b. Transferring data 

64 (2.55) 

N/A- 

c. Acceptable and unacceptable uses for shared data 

100* (3.45) 

82* (3.00) 

6. Legal aspects of data ownership and rights 

92* (3.58) 

83* (3.25) 

a. Ownership of data, patents, copyrights, and intellectual 
property 

83* (3.50) 

83* (3.08) 

b. Institutional versus research rights to own and use data 

92* (3.50) 

75* (3.08) 

c. Commercially useful data 

100* (3.58) 

75* (3.17) 

d. Negotiating contracts 

33 (2.50) 

N/A- 

7. Data privacy 

100* (3.50) 

67* (3.00) 

a. HIPAA and other privacy rules 

67* (3.50) 

58 (2.83) 

b. HIPAA and other privacy standards 

55 (2.91) 

50 (2.60) 

c. Confidentiality protection techniques 

100* (3.42) 

75* (3.00) 

8. Scientific methodology issues, including research design, objectivity, 
and bias 

92* (3.67) 

92* (3.33) 

a. Importance of research design 

100* (3.75) 

100* (3.50) 

b. Elements of good scientific design and methodology 

100* (3.75) 

100* (3.42) 

c. Proper use versus abuse of statistics 

100* (3.75) 

100* (3.45) 

d. Challenges to maintaining objectivity in designing research 
questions, controlling bias 

92* (3.58) 

92* (3.25) 

9. Data reporting 

100* (3.75) 

83* (3.17) 

a. Ethical issues when reporting data in publications 

92* (3.67) 

75* (3.08) 

b. Responsibility to interpret findings appropriately to diverse 
audience, scientific and otherwise 

100* (3.58) 

75* (2.83) 

10. Special issues related to scientific roles 

82* (3.18) 

64 (2.73) 

a. Obligations of students to supervise their own data collection 
efforts 

64 (2.91) 

N/A- 

b. Roles and relationships among team members 

92* (3.25) 

67* (2.58) 

c. who has the authority to make data related decisions 

92* (3.25) 

55 (2.55) 
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Legend; 

* = Item achieved a “consensus” by receiving a rating of important or very important from two-thirds of panelists 

^ = Not applicable because these items were eliminated after round 2 and their importance of being assessed was not 
measured 


Table 4 


Proposed Content for “MentorlTrainee Responsibilities” 


Topic (Subtopics indented) 

Percentage of panelists rating 
item as “important” or “very 
important” (Mean score) 

Teaching 

Assessing 

1. Definitions and expectations of the mentor/trainee relationship 

100* (3.75) 

36 (2.36) 

a. Defining research advisors, mentors, and trainees — across a variety of 
settings including degree programs, postdoctoral training, and jobs 

42 (2.50) 

N/A* 

b. Boundaries of the mentor/trainee relationship 

100* (3.58) 

45 (2.36) 

2. Power relationships and the potential problems they involve 

100* (3.58) 

40 (2.50) 

a. Power structures and hierarchical relationships within science and the 
mentor-trainee relationship 

92* (3.33) 

25 (2.08) 

b. Friendships and mentoring relationships 

42 (2.50) 

N/A* 

c. Harassment, sexual and other types 

67* (3.08) 

42 (2.42) 

3. Scientific responsibilities of the mentor 

100* (3.42) 

50 (2.58) 

a. Promoting professional research skills, including identifying research 
questions, writing proposals, conducting research, and publishing 

92* (3.17) 

33 (2.42) 

b. Fostering research compliance (IRB, lUCUC, etc.), RCR, and integrity 

100* (3.58) 

75* (3.08) 

c. Finding funding and negotiating grants and contracts 

33 (2.25) 

N/A* 

d. Sharing discipline-specific wisdom on how to operate in the field 

33 (2.33) 

N/A* 

4. Non-scientific responsibilities or roles of the mentor 

67* (2.92) 

42 (2.08) 

a. Career counseling, including trainees with science and non-science career 
goals 

42 (2.42) 

N/A* 

b. Conflict resolution 

67* (3.00) 

25 (2.00) 

c. Fostering autonomy with trainees while accomplishing mentor’s goals 

67* (2.92) 

25 (2.17) 

d. Management skills 

42 (2.50) 

N/A* 

5. Responsibilities of trainees within the mentor-trainee relationship 

100* (3.42) 

45 (2.45) 

a. Work with integrity 

100* (3.42) 

42 (2.42) 

b. Willingness to blow whistle or challenge misconduct and questionable 
conduct 

100* (3.50) 

36 (2.36) 

6. How to get the most out of the mentor/ trainee experience 

58 (2.67) 

N/A* 
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Topic (Subtopics indented) 

Percentage of panelists rating 
item as “important” or “very 
important” (Mean score) 

Teaching 

Assessing 

a. Optimal characteristics of mentors and trainees 

58 (2.58) 

N/A* 

b. Effective mentoring strategies and characteristics 

83* (3.08) 

27 (2.27) 

c. Contracting for a good mentoring relationship 

33 (2.33) 

N/A* 

7. Addressing challenges and problems in the mentor-trainee relationship 

100* (3.25) 

27 (2.27) 

a. Conscientious refusal 

58 (2.58) 

8 (1.83) 

b. Importance of clear communication of expectations 

100* (3.25) 

33 (2.42) 

c. Dealing with diversity of cultures, races, and other personal traits 

92* (3.25) 

42 (2.33) 


Legend: 

* = Item achieved a “consensus” by receiving a rating of important or very important from two-thirds of panelists 

^ = Not applicable because these items were eliminated after round 2 and their importance of being assessed was not 
measured 


Table 5 

Proposed Content for “Publication Practices and Responsible Authorship” 


Topic (Subtopics indented) 

Percentage of panelists rating 
item as “important” or “very 
important” (Mean score) 

Teaching 

Assessing 

1 . The significance of authorship 

91* (3.45) 

55 (2.64) 

a. The benefits of publishing 

40 (2.70) 

N/A* 

b. The problems of inappropriate authorship for legitimate authors, illegitimate 
authors, and science 

91* (3.45) 

73* (3.00) 

2. Authorship assignment 

91* (3.36) 

64 (2.73) 

a. Authorship criteria 

91* (3.55) 

64 (2.91) 

i. Substantial intellectual contribution to study or text 

100* (3.64) 

73* (3.27) 

ii. Familiarity with and approval of the final text 

82* (3.36) 

55 (2.91) 

b. Ideal of transparent contributions 

73* (3.00) 

45 (2.45) 

c. Multiple authors: how to determine senior/first author 

82* (3.36) 

55 (2.73) 

d. Appropriateness of discussing authorship at outset of a project 

91* (3.64) 

64 (3.09) 

e. Acknowledgments: purpose and examples (including faculty contributions to 
students work) 

90* (3.40) 

60 (2.90) 

f Variation of standards and norms across disciplines 

82* (3.00) 

45 (2.27) 
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Topic (Subtopics indented) 

Percentage of panelists rating 
item as “important” or “very 
important” (Mean score) 

Teaching 

Assessing 

3. Inappropriate authorship practices 

73* (3.36) 

55 (3.00) 

a. Ghost authorship 

64 (3.09) 

55 (2.73) 

b. Forced or “courtesy” authorship, e.g., when students are asked to add 
authors for political reasons 

73* (3.27) 

55 (2.82) 

4. Dealing with controversies that arise in authorship 

82* (3.36) 

55 (2.73) 

5. Scientific responsibilities of authors 

91* (3.73) 

91* (3.36) 

a. Disclosure of funding sources and other sources of potential bias 

100* (3.82) 

82* (3.36) 

b. Specification of any deviations from standard scientific practices 

91* (3.55) 

82* (3.27) 

c. Full and accurate description of methods, procedures and analytic techniques 
that allows repetition 

91* (3.64) 

82* (3.27) 

d. Citation of relevant literature without bias 

100* (3.55) 

64 (3.00) 

e. Duty to report findings accurately and completely, including reporting 
critical or negative findings (even if they are contrary to own research 
agenda) 

100* (3.73) 

82* (3.45) 

6. Poor publication practices 

91* (3.45) 

73* (2.18) 

a. Plagiarism versus proper citation or paraphrasing 

100* (3.73) 

82* (3.45) 

b. Delay in reporting for commercial reasons 

70* (2.80) 

60 (2.60) 

c. Publication bias 

100* (3.36) 

64 (2.82) 

d. Text recycling; overlapping publication; duplicate and salami publication 

100* (3.55) 

64 (2.82) 

e. Quality standards 

91* (3.27) 

64 (2.73) 

7. Protecting privacy in publication 

60 (3.00) 

N/A* 

8. Addressing compliance with ethical standards within articles (e.g., 

mentioning IRB or lACUC approval, and discussing ethically controversial 
elements of a study) 

100* (3.18) 

55 (2.64) 

9. Responsible disclosure of scientific information within the popular press 

60 (2.60) 

N/A* 


Legend: 

* = Item achieved a “consensus” by receiving a rating of important or very important from two-thirds of panelists 

^ = Not applicable because these items were eliminated after round 2 and their importance of being assessed was 
not measured 
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Table 6 


Proposed Content for “Peer Review” 


Topic (Subtopics indented) 

Percentage of panelists rating 
item as “important” or “very 
important” (Mean score) 

Teaching 

Assessing 

1 . The significance of peer review 

100* (3.64) 

73 (3.09) 

a. Peer review as a mechanism for quality assurance in publication and funding 

100* (3.18) 

55 (2.64) 

b. The need for reviewers to be competent and genuine peers 

91* (3.36) 

64 (2.82) 

2. Conflicts of Interest and Peer Reviews 

100* (3.73) 

91* (3.36) 

a. Identifying potential conflict of interest reviewers may have 

100* (3.73) 

82* (3.18) 

b. Managing conflicts of interest by excusing oneself from a review or 
disclosing and managing the conflict with the assistance of those directing the 
review 

100* (3.82) 

91* (3.27) 

c. Other sources of peer review bias 

82* (3.09) 

55 (2.64) 

3. Qualities of a good review/reviewer 

82* (3.36) 

55 (2.64) 

a. Respecting confidentiality and intellectual property (e.g., by avoiding use of 
information and destroying manuscripts after review) 

91* (3.27) 

64 (2.91) 

b. Fairness and objectivity 

91* (3.55) 

70* (3.10) 

c. Collegiality — conveying a respectful and professional tone while offering 
critical feedback 

80* (3.20) 

40 (2.30) 

d. Timeliness 

82* (3.18) 

45 (2.45) 

e. Providing clear, scientifically competent, and complete reviews 

91* (3.27) 

64 (3.00) 

4. Logistics of peer reviewing 

50 (2.40) 

N/A^ 

a. Format of written review 

30 (2.20) 

N/A^ 

b. Peer review process 

60 (2.70) 

N/A^ 

c. Selection of reviewers 

50 (2.60) 

N/A^ 

5. Responding to reviewers 

82* (3.18) 

60 (2.70) 

a. Responding to competent reviews: the revision and resubmission process 

60 (2.60) 

N/A^ 

b. Responding to questionable, biased, or conflicted reviews: the roles of 
authors (Pis) , editors, and scientific review chairs 

91* (3.18) 

64 (2.64) 

c. Inappropriate responses to reviewers and modifications to publications or 
proposals 

60 (2.50) 

N/A^ 

6. Reviewer roles in ensuring RCR 

82* (2.91) 

36 (2.27) 

7. Editorial responsibilities 

55 (2.73) 

36 (2.27) 

a. Selecting appropriate reviewers 

55 (2.73) 

36 (2.27) 

b. Attending to matters of RCR (proper authorship, disclosure of bias and 
conflicts, etc) - 2.70 

60 (2.70) 

N/A^ 

c. Respecting rights of rebuttal and mediating disputes 

60 (2.60) 

N/A^ 

d. Maintaining confidentiality 

64 (3.00) 

45 (2.55) 
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Legend; 

* = Item achieved a “consensus” by receiving a rating of important or very important from two-thirds of panelists 

^ = Not applicable because these items were eliminated after round 2 and their importance of being assessed was 
not measured 


Table 7 

Proposed Content for “Collaborative Science” 


Topic (Subtopics indented) 

Percentage of panelists rating 
item as “important” or “very 
important” (Mean score) 

Teaching 

Assessing 

1 . The nature and advantages of successful collaborations 

83* (3.17) 

50 (2.50) 

a. Reasons for collaborating 

58 (2.83) 

N/A* 

b. Risks and benefits of collaborations 

75* (3.08) 

42 (2.17) 

c. Identifying a good collaborator 

83* (3.08) 

33 (2.33) 

2. Types of collaboration 

63 (2.73) 

22 (2.00) 

a. Collaboration within an institution 

67* (2.75) 

17 (2.00) 

b. Collaboration between institutions 

58 (2.67) 

8 (1.83) 

c. International collaboration 

58 (2.83) 

N/A* 

3. Working well with others 

92* (3.25) 

27 (2.27) 

a. Identifying the authority and procedures for establishing collaborative 
relationships 

92* (3.00) 

33 (2.25) 

b. Defining and clarifying roles, responsibilities, and expectations in a 
collaboration 

100* (3.42) 

33 (2.42) 

c. Identifying mechanisms for ongoing decision-making 

75* (2.92) 

25 (2.17) 

d. When are written agreements necessary, and what should be addressed in 
contracts 

92* (3.25) 

75* (2.75) 

e. Knowing how and when to end collaborative relationships 

83* (3.00) 

33 (2.17) 

4. Dealing with challenges in collaborative relationships 

100* (3.40) 

40 (2.50) 

a. Addressing failures in RCR or research integrity 

83* (3.33) 

82* (2.73) 

b. Allocating rewards such as credit, authorship, ownership, and rights of use 

100* (3.58) 

83* (3.08) 

c. Dealing with competition 

50 (2.58) 

N/A* 

d. Addressing power discrepancies when junior scientists collaborate with 
senior scientists 

75* (3.00) 

50 (2.58) 

5. The role of institutions in collaborative science 

58 (2.67) 

N/A* 

a. Working with appropriate officers 

50 (2.58) 

N/A* 

b. Knowledge of institutional policies 

83* (3.08) 

50 (2.58) 
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Legend: 

* = Item achieved a “consensus” by receiving a rating of important or very important from two-thirds of panelists 

^ = Not applicable because these items were eliminated after round 2 and their importance of being assessed was 
not measured 


Table 8 


Proposed Content for “Research Misconduct” 


Topic (Subtopics indented) 

Percentage of panelists rating 
item as “important” or “very 
important” (Mean score) 

Teaching 

Assessing 

1 . Significance of misconduct 

100* (4.00) 

100* (3.75) 

a. History of scientific misconduct 

82* (3.00) 

42 (2.17) 

b. Incidence rate of misconduct 

58 (2.58) 

N/A* 

c. Consequences of misconduct for individuals, laboratories, science, and 
society 

100* (3.64) 

67* (3.00) 

2. Factors that contribute to scientific misconduct 

100* (3.73) 

75* (3.25) 

a. Effects of laboratory environment 

100* (3.64) 

75* (3.08) 

b. Reward systems in academic and industry settings 

100* (3.45) 

67* (2.83) 

3. Plagiarism 

100* (3.91) 

83* (3.33) 

a. Definition and examples 

100* (3.73) 

92* (3.25) 

b. Case studies with outcomes and punishments 

83* (3.18) 

58 (2.67) 

4. Falsification 

100* (4.00) 

92* (3.50) 

a. Definition and examples 

100* (3.82) 

92* (3.25) 

b. Case studies with outcomes and punishments 

100* (3.60) 

73* (3.00) 

5. Fabrication 

100* (4.00) 

92* (3.50) 

a. Definition and examples 

100* (3.82) 

91* (3.18) 

b. Case studies with outcomes and punishments 

100* (3.55) 

75* (3.00) 

6. Other serious deviations from scientific best practices 

80* (3.22) 

60 (2.70) 

a. Sabotage 

58 (3.00) 

N/A* 

b. Questionable research practices (e.g., data manipulation) 

100* (3.55) 

75* (3.00) 

c. Unintentional deviations 

100* (3.45) 

67* (2.92) 

7. Regulations and policies addressing misconduct 

82* (3.40) 

82* (3.18) 

a. The Office of Research Integrity’s role in addressing misconduct 

92* (3.18) 

50 (2.58) 

b. Institutional policies 

92* (3.36) 

67* (2.92) 

8. Responding to observed misconduct 

100* (3.91) 

92* (3.42) 
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Topic (Subtopics indented) 

Percentage of panelists rating 
item as “important” or “very 
important” (Mean score) 

Teaching 

Assessing 

a. Evidence needed to report misconduct 

100* (3.73) 

64 (3.00) 

b. Whistle blowing, including responsibilities and protections for whistle 
blowers 

100* (3.82) 

92* (3.25) 

c. Alternatives to whistle blowing with illustrations of good and bad responses 

92* (3.45) 

75* (2.92) 

9. Studying taboo, controversial, or politically sensitive research topics 

83* (3.09) 

50 (2.50) 


Legend: 

* = Item achieved a “consensus” by receiving a rating of important or very important from two-thirds of panelists 

^ = Not applicable because these items were eliminated after round 2 and their importance of being assessed was 
not measured 


Table 9 

Proposed Content for “Conflicts of Interest and Commitment” 


Topic (Subtopics indented) 

Percentage of panelists rating 
item as “important” or “very 
important” (Mean score) 

Teaching 

Assessing 

1 . The significance of conflicts of interest 

100* (3.73) 

75* (2.83) 

a. Historical examples of conflicts of interest in science 

50 (2.58) 

N/A^ 

b. Psychology and conflicts of interest, i.e., how conflicts of interest may cloud 
judgment or influence decisions 

83* (3.33) 

50 (2.42) 

c. The pervasiveness of conflicts of interest, including sponsored research 

83* (3.25) 

33 (2.17) 

d. Consequences of conflicts for researchers, institutions, students and research 
participants 

92* (3.33) 

33 (2.33) 

e. Why conflicts of interest are pervasive and not always bad 

83* (3.42) 

50 (2.50) 

2. Types, definitions, and examples of conflicts of interest 

100* (3.55) 

50 (2.50) 

a. Financial conflicts of interest, including gifts and honoraria, patents, spin 
off companies, SBIR/STTRs, personal investments, funding contracts with 
industry 

92* (3.58) 

67* (3.00) 

b. Non-fmancial conflicts of interest (e.g., recognition, publications, 
promotions) 

58 (2.67) 

N/A^ 

c. Role conflicts (e.g., physician-researcher or teacher-researcher) and 
conflicting duties to self clients, institutions and society 

75* (3.08) 

50 (2.42) 

d. Conflicts of interest are objective relationships — they do not imply actual or 
intended wrong doing 

83* (3.00) 

33 (2.25) 
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Topic (Subtopics indented) 

Percentage of panelists rating 
item as “important” or “very 
important” (Mean score) 

Teaching 

Assessing 

3. Conflicts of commitment (i.e., dividing one’s percent effort within a job) — 
definition, examples, and management 

58 (2.67) 

N/A^ 

a. Effort reporting rules 

58 (2.67) 

N/A^ 

b. Balancing sponsored research with other duties 

58 (2.67) 

N/A^ 

c. The perils of becoming over extended 

67* (2.92) 

33 (2.17) 

4. Institutional conflicts of interest 

50 (2.58) 

N/A^ 

a. Conflicted oversight (e.g., IRB and lACUC members are employees who 
review work of peers) 

42 (2.50) 

N/A^ 

b. Institutional investments and profits from research 

42 (2.42) 

N/A^ 

5. Managing conflicts of interest 

100* (3.50) 

67* (2.83) 

a. Avoiding or eliminating conflicts of interest 

100* (3.25) 

58 (2.75) 

b. Disclosing conflicts of interest / conflicts of interest and informed consent 

100* (3.58) 

83* (3.00) 

c. Management plans, including, e.g., role separation 

42 (2.67) 

N/A^ 

6. Conflicts of interest law and policy 

50 (2.58) 

N/A^ 

a. Regulatory and statutory laws 

50 (2.75) 

N/A^ 

b. Institutional policies on conflicts of interest 

82* (3.27) 

55 (2.55) 


Legend: 

* = Item achieved a “consensus” by receiving a rating of important or very important from two-thirds of panelists 


^ = Not applicable because these items were eliminated after round 2 and their importance of being assessed was 
not measured 
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